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Abstract 

This article describes the purpose, development, and implementation of a cognitive-based instructional 
intervention and its impact on learning motivation. The study was conducted in a programming-based problem- 
solving course for first -year engineering students. The results suggest that the instructional intervention 
developed based on the hierarchical analysis of intellectual skills development and partial-to-whole learning 
task approach has significant correlation with the Satisfaction component of ARCS Motivational Design Model. 

Introduction 

Problems with current approaches to study of motivation 

Motivation is a critical factor in learning (Linnenbrink & Pintrich, 2002) and many researchers have 
examined the subject from various perspectives (Gonzalez- Pienda, et al., 2002; Hancock, 2002; Cokley, et al., 
2001). Three issues trouble current studies of learning motivation: the lack of a systematic approach, neglect of 
other aspects of learning, and the inability to separate different aspects of motivation. 

The affective and complex nature of motivation makes it is difficult to study, resulting in many 
motivational studies that lack a systematic approach. This lack of sytematicity limits reproduction of the effect 
and makes application difficult in practical situations. A review of motivational studies based on various aspects 
of learning (e.g. cognitive, social, and attitudinal) reveals that there is no sound systematic approach applied to 
them; most are solely based on motivational theories (Harlen & Crick, 2004) and frameworks to establish a 
broad, inductive goal without identifying a motivational problem. Identifying the motivational problem can 
clarify the motivational analysis and intervention. For example, Bandura’s Self-Regulation theory is well 
received in the learning motivation field and efforts are being made to identify the measurable variables 
affecting levels of Self -Regulation (Miller & Brickman, 2004). However, there is no suggestion as to how those 
variables or components can be practically applied to resolve specific problems. 

In other words, one problem with learning motivation studies is a lack of linkage between theories and 
practical instructional practices. Many theorists have proposed theories on how people get motivated and what 
behaviors can be stimulated by motivation (Weiner, 1985; Ames, 1992; Anderman & Maehr, 1994; Bandura, 
1997). However, educators are still in search of practical and applicable guidelines that would enable them to 
convert motivation-enhancing theories into practical instructional practices with confidence (Hancock, 2002). 

Another problem with current studies of learning motivation is their narrow approach. Given the 
complexity of investigating and measuring learning motivation in any instructional setting, various aspects of 
learning should be considered when trying to address motivational issues holistically. Aspects ranging from 
internal factors like student learning styles to external factors like the learning environment and applied 
instructional strategies should be considered. 

Finally, the other learning components, such as instructional strategies, along with affective 
components traditionally associated with motivation such as enthusiasm and understanding of content, need to 
be separated and their effects on each other evaluated. This poses the concern of how reliable and valid 
motivational studies are if multiple issues are addressed simultaneously. Because factors affecting learning do 
interact with each other, a confounding effect can occur. Motivation does not occur in a vacuum and the effects 
other factors in learning have on motivation are traditionally neglected (Astleitner & Wiesner, 2004). 

A Systematic Approach: Selection of the ARCS Model 

In order for the results of motivational studies to be more practical in addressing known instructional 



374 




or learning problems, perhaps a problem-solving process should accompany the investigation process. A 
generic instructional design process whose components are found in many instructional design models (e.g. 

Dick, 1996; Smith & Ragan, 1993; Gentry, 1994, Gustafson & Branch, 1997) may be an appropriate tool. The 
ADDIE process is geared towards solving instructional problems in five general steps; (A) analysis of the 
instructional problem, (D) design and (D) development of the instructional intervention, (I) implementation of 
the intervention, and (E) evaluation of the outcome(s). These systematic steps can be applied to the 
investigation of learning motivation due to their generalizability. 

A motivational model with practical instruments should be used along with a problem-solving process 
to address specific aspects of motivation that a broad instructional design model like ADDIE cannot. The ARCS 
model, a motivational design model (Keller, 1987a, b), provides both a theoretical framework and a tool for 
assessing motivational levels by following the steps of ADDIE The ARCS model suggests that learning 
motivation is influenced by four components: Attention, Relevance, Confidence, and Satisfaction. By enhancing 
an individual component using specific motivational strategies and instructional methods, students’ learning 
motivation can be improved. Keller (1987b) suggested a systematic approach to first identify motivational 
problems and then prescribe motivational strategies to solve them. A pre - motivational survey is needed to 
establish students’ initial motivational level. Keller’s (1993) Instructional Materials Motivational Survey 
(IMMS) was developed as a situated measuring instrument to gauge the learning motivation of specific 
instructional materials. From the survey, motivational strategies can be prescribed according to identified 
motivational objectives. A post-motivational survey should be conducted to examine the effectiveness of the 
motivational strategies applied. 

Motivational Studies Focusing on Cognitive Information Processing 

Another way to make motivational studies more widely applicable is to focus on the cognitive side of 
learning. Studies on the cognitive aspects of learning, including cognition development, have established a 
scientific methodology to empirically investigate the human learning process from an information processing 
viewpoint. This deductive approach makes the studies more replicable, the focus of research questions more 
clear, and the outcomes more reliable and easier to identify. 

In addition to the measurable outcomes these scientific research designs can produce, various studies 
have suggested a possible relationship between cognitive information processing and learning motivation 
(Wolters, 2004; Chalupa, Chen, & Charles, 2001). For example, Malone and Lepper (1987) proposed four 
components to establish learning motivation, all derived from human mental cognition. The association between 
motivation and cognition is also supported by the Expectancy Theory (Vroom, 1964) and the Control Theory 
(Klein, 1989) in which the learner’s goal setting behaviors and perceived control are emphasized. Astleitner and 
Wiesner (2004) further proposed an integrated model of multimedia learning and motivation that describes the 
relationship between memory capacity and resource management, and their effect on learning motivation. 

From an instructional design viewpoint, the systematic process to approach motivational problems and 
the relationship between cognitive processing and motivation means it is possible to design and develop 
interventions for motivational problems based on a cognitively based instructional strategy. 

Purpose of the Study 

The purposes of this study are to examine the feasibility of adopting a systematic and instructional 
design-oriented research design for the investigation of learning motivation, and to investigate the relationship 
between a cognitively based intervention and learners’ motivation. This study employed (1) a procedure to 
diagnose, analyze, and develop instructional interventions for better motivational outcomes, (2) a hierarchical 
analysis of intellectual skills to better understand the complexity of the subject’s cognitive learning tasks, and 
(3) the cognitive load theory (Sweller, 1994) as the foundation for developing the intervention in response to the 
identified motivational problems. 



Methodology 



Setting and Participants 

Our focus in this study was on how subjects used a computer-based tutorial called M-Tutor™. M- 
Tutor was designed to help students learn MATLAB® syntax. MATLAB® is a computational software package 
that integrates mathematical computing, visualization, and a powerful comp uter programming language to 
provide a flexible environment for technical computing (Mathworks, 2003). 
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In the Fall semester of 2002, first-year engineering students who enrolled in ENGR 106, Engineering 
Problem-Solving and Computer Tools, were instructed to use M-Tutor as their primary means of learning 
MATLAB syntax. This 2 credit-hour, required course is designed to develop first-year engineering students’ 
abilities to solve engineering problems with appropriate computer tools. In order to accomplish the course 
objectives, the instructor needed to create an active learning environment in the lecture, which emphasizes 
fundamental engineering concepts and problem-solving strategies. However, the students needed to 
simultaneously learn to be efficient and effective users of computer tools and use those tools to solve 
engineering problems. The instructor struggled with devoting class time to simply telling students about 
computer tools and how they worked versus covering fundamental engineering concepts and how to use these 
computer tools to solve realistic problems. Thus the instructor adopted M-Tutor to help students learn 
MATLAB syntax outside of class. Students’ motivation for using M-Tutor to learn MATLAB was the focus of 
this study. 

Keller’s ARCS Model of Motivational Design and Instructional Materials Motivational Survey 
(IMMS) were adopted as the theoretical framework of learning motivation as well as the basis for quantitative 
and qualitative data collection. The study employed the one group pre and post-tests design. Three surveys were 
implemented: Pre -motivational survey, post-motivational survey, and an additional survey on the use of 
student -made glossary. The treatment was having students complete instructor-guided glossary items. The last 
survey focused on the glossary itself was implemented in order to associate students’ perception towards the use 
of the glossary with their learning motivation levels. The pre -motivational survey based on the ARCS Model 
indicated that students gave a relatively low rating on the Satisfaction component. Students’ qualitative 
responses also suggested that they felt frustrated when using the computer-based tutorial. The main reason for 
that reaction is the lack of connection between isolated coding tasks and their application to engineering 
problems. Thus the motivational strategy was developed based on the pre -motivational survey with specific 
emphasis on the Satisfaction component of ARCS Model. 

The strategy for enhancing students’ Satisfaction level is to provide students with more opportunities 
to gain a sense of accomplishment on course assignments by using M-Tutor™. Providing immediate feedback 
is also considered a crucial element for better motivational outcome. Thus the student -made glossary 
assignment was developed as the intervention to carry out the motivational strategy, which allows students to 
receive meaningful feedback as well as obtain a sense of accomplishment prior to solving application problems. 
The feeling of accomplishment is obtained by allowing students to go through smaller parts of the learning task 
(i.e. individual syntax as opposed to a chunk of coding). There are five columns on the glossary form: (1) 
syntax, (2) overview of the syntax, (3) student developed test case(s) based on instructor guidelines, (4) hand 
computations to predict results of test cases, and (5) MATLAB results. The complexity level of each column is 
increased by following the hierarchical analysis of intellectual skills (i.e. discrimination, concepts, and rules). 

By accomplishing the lower levels allows students to build a schema and tie the ideas together, so that when 
they are presented with a problem, they can relate the problem to their new schema. 

Each glossary item was designed by the instructor according to immediate assignments since it helps 
students to better transfer newly acquired programming skills (composing with syntax) to application problems. 
Students were asked to complete the glossary before they worked on application problems . 

Research Design 

This study included all five steps of the generic instructional design process discussed earlier (i.e. 

ADDIE). 

(1) The analysis of the pre -motivational survey and examining its data. 

(2) The design of motivational intervention based on the pre -motivational data. 

(3) The development of a motivational intervention based on previous relevant research and theories as 
well as subject matter experts’ input. 

(4) The implementation of the developed motivational intervention (instructional strategy) to the actual 
instructional setting. 

(5) The evaluation of the motivational strategy by a post-motivational survey and an intervention-specific 
survey. 

Development of Motivational Strategies and Data Processing 

Pre- and Post-Motivational Survey and Data Analysis The first motivational survey was conducted in 
the week after the first M-Tutor assignment while the post-motivational survey was given after the 
implementation of intervention. The survey instrument collected students’ Pre- and Post- reactions towards the 
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tutorial. Keller’s IMMS survey was adapted in order to accommodate the computer-based study setting. It 
consisted of 36 rating questions (Cronbach’s Alpha= 0.917). The Instructional Materials Motivational Survey 
(IMMS) (Keller, 1993) was developed based on the ARCS Model. The IMMS consists of 36 statements that are 
rated using Likert -type scales (1 = Not True; 5 = Very True). Each item is mapped to an individual ARCS 
component and provides a measure of the respondent’s perception of that particular component. Quantitative 
data are composed of ratings from the IMMS. It is important to re member that responses to a Likert -type scale 
item generate categorical data that cannot be averaged to provide a mean response for an individual survey item 
Therefore, the frequency with which students responded "Mostly True" and "Very True" on individual survey 
items were computed. For all items mapped to a particular ARCS component, the frequency with which 
students responded "Mostly True" (4) and "Very True" (5) was used to provide a single quantitative measure of 
that ARCS component. Qualitative data was also collected by open-ended questions attached to the survey. The 
main purpose of collecting qualitative data was to better identify design issues within each instructional 
component. 

Studies indicate that the ARCS Model is applicable in the computer-based or web-based instructional 
environment (Keller, 1999; Keller & Song, 1999; Knowlton , Shellnut & Savage, 1999; Park& Hannifin, 1993) 
although it was originally designed for developing motivating instructional materials in traditional face-to-face, 
classroom settings. For this pilot study, the IMMS was modified to assess the motivational effectiveness of M- 
Tutor. Each survey item was revisited and, as needed, re -focused on the research question, which was to 
diagnose students’ motivational level in using the tutorial as a learning tool, with the expectation that students 
would effectively learn MATLAB syntax and effectively use MATLAB as a tool for solving engineering 
problems. 

Design, Development, Implementation, and Evaluation of Motivational Strategies After the pre- 
motivational survey, we analyzed the quantitative as well qualitative data to determine which aspect of 
motivation to address. A coding system (Table 1) was developed for analyzing qualitative data based on design 
principles of multimedia courseware (Szabo & Kanuka,1998; Evans & Edwards, 1999; Coscarelli & Shrock, 
2000). The instructional coding system categorizes qualitative responses from each mapped ARCS Model open- 
ended question into various instructional components . The instructional components involved were coded as 
interface design, content, learning support, and implementation. Each qualitative response could be coded in 
one or more categories. The research direction was visited repeatedly during the development of the coding 
system to insure the validity of items (Coscarelli & Shrock, 2000). 

By triangulating the quantitative and qualitative data, valuable information can be gained. First, the 
research team can identify which instructional component(s) are most influential on students’ motivational 
levels. Second, the research team can map the instructional components of the tutorial (interface, content, 
learning support, and implementation) to the ARCS Model components (Attention, Relevance, Confidence, and 
Satisfaction). 

Also to explicitly evaluate the instructional intervention, a ten-question survey was developed to 
measure students’ reactions towards the intervention independently of its effectiveness in enhancing their 
motivational level. (Cronbach’s Alpha =0.926, N=957). 

Table 1. EXAMPLES OF CODES AND INTERPRETATIONS 



Instructional 
Component Code 


Keyword Examples 


Interface Design 


Text, graphs, navigational system, interactivity 
menu 


Content 


Relevance, easiness and difficulty of information 


Learning Support 


Textbook, assistance, Help session, feedback, 
exercises 


Implementation 


Course structure, technical infrastructure 
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Results 



Pre-motivational survey analysis 

Table 2 indicates that students initially provided the least of amount of positive response towards the 
Satisfaction component of ARCS model based on the frequencies of “Mostly True” and “Very True” that were 
selected for each component. Additionally, the qualitative data suggested a considerable amount of students did 
not perceive using the tutorial as a satisfactory experience. Therefore the objective, based on the pre- analysis , 
was to increase students’ satisfaction level towards the tutorial with a feasible instructional intervention. 



TABLE 2. Comparison of frequencies of "Very True ” selected by participants among ARCS components 



Pre -Motivational Survey 


Attention 


Relevance 


Confidence 


Satisfaction 


Frequency of response indicating 
“Mostly True’’ and “Very True” 


2692 


3568 


2749 


711 


combined 











Identifying Motivational Strategies 

The qualitative data analysis from the pre-motivational survey indicated that the main issue influencing 
students ’ motivation was the transfer between learning programming syntax and using the syntax in problem- 
solving scenarios. The problem of transferability is common in abstract cognitive skill instruction such as 
programming or mathematics. Quilici and Mayer (2002) conducted a study on the transferability of statistical 
skills to word problems , which suggests the transfer of knowledge can be facilitated by providing systematic 
and frequent training. Renkl et al. (2002) also suggests that by providing problem-solving examples with 
gradually increasing levels of difficulty students were able to develop complete problem-solving skills by 
themselves. The low ratings on the Satisfaction component provided further support for designing 
corresponding motivational strategies. In order to enhance the Satisfaction level, students should have more 
opportunities to receive feedback and have feelings of accomplishment by exercising newly -learned knowledge 
(Keller, 1987b). As a result, a student -made glossary was developed (Table 3), which allowed students to define 
and practice MATLAB syntax prior to being given engineering problems to solve. 



TABLE 3. Example of student-made glossary 



MATLAB 

Syntax 


Overview of Syntax 


Test Case(s) 


Expected Result 
for Each Test 
Case 


MATLAB Command Used 
for Each Test Case and 
Result 


who 


Description: Displays 
a list of all the 
variables available in 
the workspace. 

Input Arguments: 

none 

General Format: 

» who 


Enter these 
MATLAB 
commands: 
»C = 0 
» x = 3 
» who 


Lists the 
variables C and x 
(and anything 
else previously 
created) as being 
available in the 
workspace 


As given under Test Cases. 

Result: 

Your variables are: 

C x 


save 


Description: Saves 
all results, or specified 
results, to a MAT file 
Input Arguments: 
filename and specific 
variables (optional) 

General Format: 

»save filename 
»save filename varl 


1. Create three 
variables. 

2. Save all of the 
variables you've 
created in a 
filename called 
vars_all. 

3. Save two of 
the three 
variables you 


Saves variables 
to a MAT file 

Where are the 
files saved? Give 
the exact path 
name. 

To the current 
directory which 
should be h:/Mv 


X = 1 
Y = 2 
Z = 3 

save vars_all 
save vars_two X Y 
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MATLAB 

Syntax 


Overview of Syntax 


Test Case(s) 


Expected Result 
for Each Test 
Case 


MATLAB Command Used 
for Each Test Case and 
Result 




var2 ... 


created in a 
filename called 
vars_two. 


Documents/m- 

tutor 





Implementation of Motivational Strategy: The Student-Made Glossary 

The student-made glossary was implemented as a homework assignment towards the end of the 
semester. The glossary items came from the last M -Tutor learning unit assigned and included syntax needed to 
complete problem-solving assignments for the remainder of the semester. Students were asked to fill out all four 
columns as the first part of the assignment. Later the commands seen in the glossary were applied to other 
homework problems. The columns conform to Renkl’s idea (Renkl, et ah, 2002) of increasing difficulty. In the 
first column, the student describes the programming syntax in terms of how and why to use it. In the next 
column, the student must generate a small sample case using the syntax. In the next two columns, the student 
generates an answer without the computer, then feeds their sample code into the MATLAB program and 
records its response. 

Post-Motivational Survey Data Analysis 

The same instrument (IMMS) used in the pre -motivational survey was administrated to conduct the 
post-motivational survey to make a valid comparison, as shown in Table 4. 

A one-way ANOVA was applied to data analysis to investigate the significance of variance between 
pre - and post-motivational surveys as the result of implementing the motivational intervention, that is, the 
student -made glossary. 

The analysis indicated that the use of the glossary significantly lowered students’ satisfaction as shown 
in Table 4 based on the aforementioned frequency method, which contradicts our hypothesis that the 
satisfaction level would increase when students were provided with opportunities to practice newly acquired 
skills and to gain meaningful feedback. Possible reasons contributing to the result will be addressed in the 
Discussion section. 



TABLE 4. One-way ANOVA on students ' satisfaction towards M-Tutor between Pre- and Post -motivational 
surveys 





Pre Mean 


Post Mean 


F 


Significance level 


N= 957 


0.75 


0.57 


10.717 


0.001 



Post-Motivational Survey and Glossary Survey 

The purpose of implementing the glossary survey (Cronbach’s Alpha= 0.926) was to collect data on 
students’ reactions to using it as a supplementary learning tool and its relation to students ’ satisfaction level on 
the post-motivational survey. To thoroughly evaluate the implemented motivational strategy (i.e. student-made 
glossary), it is important to measure students ’ initial reaction towards the glossary as the first level of evaluation 
(Kirkpatrick, 1998). Interestingly the linear regression analysis suggests there is a significant relationship 
between students’ reactions towards the glossary and the post-satisfaction response frequencies as shown in 
Table 5. Students’ reactions towards the glossary, which was developed based on the different levels of 
intellectual skill hierarchical analysis, had a statistically significant relationship with students’ post-satisfaction 
level as defined by Keller’s ARCS Model. 

TABLE 5. Linear regression analysis between students’ reaction towards glossary and Post-satisfaction 
responses frequencies 



Y=aX+b 


Coefficient 


t statistic 


P value 


R Square 


a( Glossary Reaction) 


0.1103 


9.2985 


9.42E-20 


0.083 



The study suggests that the cognitively based instructional intervention (i.e. student -made glossary) is 
influential on the variance of students’ Satisfaction level in a computer-based instructional setting, though the 
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effect is considered negative towards students’ overall motivational gain. Further discussion will explore the 
underlying reasons as to how the construction of their problem-solving schema affects students’ perceived 
motivational levels during the development of higher cognitive skills. 

As the result of evaluation for the implemented intervention, the significant linear regression relationship 
between students’ positive reaction to glossary and their post-satisfaction level suggests, interestingly, the 
possibility of other factors affecting the motivational gain induced by the glossary, which also will be discussed 
in the next section. 



Discussion 

Traditionally motivational studies have not been conducted very systematically, perhaps due to the 
complex nature of motivation and the difficulty in measuring it. This study uses a generic systematic model, 
ADD1E, often used in instructional design, with the following steps: analysis, design, development, 
implementation, and evaluation. The ADD1E model is general enough to approach most problems, including 
motivation. The appearance of each component of ADD1E in most of the accepted instructional design models 
today is a sign of its theoretical grounding. 

This study focused primarily on the Satisfaction component after the pre- motivational survey, and 
showed significant change after the intervention. Students felt less satisfied towards M-Tutor after the use of the 
glossary. However, the regressive relationship between students’ positive reaction to the glossary (i.e. how they 
like the intervention) and their post-satisfaction level suggests otherwise. Before we delve into the reasons as to 
why the loss of motivational gain, it is necessary to first to understand the function of student -made glossary 
and how it is compatible with Keller’s definition of the Satisfaction component. Keller breaks Satisfaction up 
into three parts: natural consequences, positive consequences, and equity (Keller, 1987). 

It could be that either positive consequences or equity relate to this study. The design and development 
of student-made glossary, however, focused on natural consequences mainly because it can give students a 
meaningful way to transfer the information they have learned. The student-made glossary is a cognitive advance 
organizer that addresses that aspect of satisfaction by helping student create a glossary of terms, leading them 
from knowing what the term is to practicing how to use it. The organizer categorizes students’ knowledge, then 
provides a way for students to use it in a meaningful way in context and therefore meeting the "natural 
consequences" requirement of satisfaction. 

Given the pedagogically -sound theoretical foundation and rationale of designing and developing the 
student -made glossary derived from either cognitive loading theory (e.g. worked example) (Sweller, 1994) or 
the ARCS motivational design model (Keller, 1987a, b), the negative effect on motivational gain induced by the 
glossary raises issues not only associated with the developmental portion of instructional intervention, but 
redirects our attention to the overall research design, especially in the implementation part of the study. 

The pre -motivational survey was administered in the first quarter of the semester while the 
instructional intervention (i.e. student -made glossary) was not in place until the last quarter due to time needed 
for the pre -motivational survey data analysis and the design and development of the glossary. This temporal gap 
is typical in the developmental research (Richey& Nelson, 1996). Additionally the study was conducted 
simultaneously with the progression of the course. Therefore it is very likely that the glossary is not the cause of 
the motivational loss. Various factors such as course structure, complexity of learning tasks, interaction between 
teaching staff (i.e. professors and teaching assistants) and students , and the utilization of M-Tutor could have 
influenced students ’ motivation levels. It is questionable whether the student-made glossary could overcome the 
interactions among the aforementioned factors in terms of students ’ motivational gain or loss. In order to better 
measure any instructional intervention’s effectiveness especially on perceived motivation, a more controlled 
study setting (e.g. smaller sample size) with a much shorter time gap between pre - and post-motivational 
surveys needs to be applied. 



Conclusion 

This study presents the concept and feasibility of using a generic instructional design process (ADDIE 
model) to motivational evaluation. It also demonstrates the use of the ARCS Model of Motivational Design for 
analysis and evaluation of students’ learning motivation when using an existing computer-based tutorial. 
Triangulation of quantitative and qualitative data provided insight on the impact of instructional components on 
students’ learning motivation as represented by four motivational components , which is also helpful for future 
revision of the computer-based tutorial. Finally the issue of effective implementation of an instructional 
intervention for motivational studies poses important considerations when migrating from the realmof 
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developmental research to amore replicable research design. 
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